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DETAILED ACTION 
Response to Arguments 

1 . Applicant's arguments with respect to claims 26-28, 33-43, and 49-74 have been 
considered but are moot in view of the new ground(s) of rejection. 

Claim Objections 

2. Claims 26, 36, 40, 66, and 67 are objected to because of the following 
informalities: 

Claim 26 recites the limitation "said timing information" in line 2. There is 
insufficient antecedent basis for this limitation in the claim. For the rest of this Office 
Action, "said timing information" will be treated as "said timing data." 

Claim 36 recites the limitation "said mask data" in line 2. There is insufficient 
antecedent basis for this limitation in the claim. For the rest of this Office Action, "said 
mask data" will be treated as "mask data." 

Claim 40 recites the limitation "said mask data" in line 2. There is insufficient 
antecedent basis for this limitation in the claim. For the rest of this Office Action, "said 
mask data" will be treated as "mask data." 

Claim 66 recites the limitation "said annotation data" in line 2. There is 
insufficient antecedent basis for this limitation in the claim. For the rest of this Office 
Action, "said annotation data" will be treated as "annotation data." 

Claim 67 recites the limitation "said image overlay data" in line 2. There is 
insufficient antecedent basis for this limitation in the claim. For the rest of this Office 
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Action, "said innage overlay data" will be treated as "said data for overlaying a graphics 
image." 

Appropriate correction is required. 

Claim Rejections - 35 USC § 103 

3. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

4. Claims 26-28, 33-43, 49-56, and 58-63 are rejected under 35 U.S.C. 103(a) as 
being unpatentable over U.S. Patent 6,615,408 to Kaiser et al. (Kaiser) in view of U.S. 
Patent 6,496,981 to Wistendahl et al. (Wistendahl). 

Regarding claim 28, Kaiser teaches a reproduction apparatus (fig. 1, label 1300) 
in communication with a broadcast channel (col. 4, II. 59-67; col. 5, II. 22-31), a display 
(fig. 1 , label 1200) (claimed display device). Kaiser teaches a receiver decoding a 
digital signal to recover a video signal (such as high definition television formats, see 
col. 5, II. 30) and trigger/placement zones (claimed annotation data) (fig. 2). Kaiser 
teaches a placement zone comprising the one or more frames comprising an image 
referencing a product data (col. 6, II. 19-23); Kaiser receives the overlay of a graphic on 
a video frame from accessing a data network 1400 (col. 8, II. 14-23), Kaiser is silent on 
the graphics data being included in the broadcast signal. 
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In analogous art, Wistendahl teaches inserting the N data (for mapping the 
locations of the hot spots - see col. 6, II. 27-31 , col. 6, II. 42-59) along with the media 
content (col. 3, II. 32-45, col. 7, II. 28-52), which equates to the graphics data being 
included in the broadcast signal. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Kaiser by having graphics data being included in 
the broadcast signal as taught by Wistendahl in order to efficiently enable the user to 
access additional services along with receiving media. 

Kaiser teaches placement zones, which can be viewed over one or more frames 
(col. 6, II. 19-23), fails to explicitly disclose drawing the graphics images in response to a 
viewer request and the drawing of the graphic image being synchronized to the 
underlying frame based on timing data. 

In analogous art, Wistendahl teaches drawing halos H (see fig. 7a) by 
highlighting the hotspots (col. 15, II. 1-6, col. 15, II. 54-56), which equates to in response 
to a viewer request, wherein the hot spots are viewed over one or more frames and the 
graphic image is synchronized to the underlying frame based on timing data of the 
graphic image, in that the hot spot is for a given frame and each frame has a unique 
time position in the sequence (col. 5, II. 67-3, col. 6, II. 42-46). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Kaiser by drawing the graphics images in 
response to a viewer request and the drawing of the graphic image being synchronized 
to the underlying frame based on timing data as taught by Wistendahl in order to 
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provide for synchronization of the overlays over a plurality of frames, thereby increasing 
the effectiveness of the hotspot by properly following the image. 

Regarding claim 26, the combination of Kaiser and Wistendahl teaches timing 
information comprising one of a timestamp, timecode, frame numbering, or global time 
of day (Wistendahl: col. 5-6, II. 66-3. col. 6, II. 27-31, col. 6, II. 60-63). 

Regarding claim 27, Kaiser clearly synchronizes the placement zones and the 
video in response to timing information in order to place the placement zones over the 
proper location, such as a car as shown in figures 6A-6D, see also Wistendahl: col. 5-6, 
II. 66-3, col. 6, II. 27-31, col. 6, II. 60-63. 

Regarding claim 33, Kaiser teaches product purchase actions (col. 12, II. 29-54), 
which reads on information regarding goods and services. 

Regarding claim 34, the combination of Kaiser and Wistendahl teaches 
annotation data comprising a movie trivia information, which reads on a non-commercial 
information (Wistendahl: col. 9, II. 28-39) 

Regarding claim 35 and 36, the combination of Kaiser and Wistendahl teaches a 
placement zone which references a product being displayed, wherein the placement 
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zone is a location of an object (Wistendahl: col. 5-6, 11. 66-3. col. 6, II. 27-31, col. 6, II. 
60-63). 

Regarding claims 37-39, Kaiser teaches placement zones for various frames (fig. 
2, col. 6, II. 9-17), wherein the placement can track an image at the upper left corner 
(col. 10, II. 34-38). Whereas Kaiser is silent on a location reference at the centroid pixel, 
Official Notice is taken that a center position is well known in the art, such as tracking an 
object by the center position. Therefore, it would have been obvious to one of ordinary 
skill in the art at the time the invention was made to modify Kaiser by using a center 
position (as a reference point) in order to properly track an image thereby increasing the 
effectiveness of the placement zone. 

Regarding claims 40-42, Kaiser teaches location and shape information such as 
the shape and location of the car (col. 10, II. 34-38), see figures 6A-6D, wherein the 
visual highlight (fig. 6B, label 6500) is a graphical overlay and has an outline of the car. 

Regarding claim 43, Kaiser is silent on a mathematical representation of set of 
pixels. Official Notice is taken that mathematical representation of pixels are well known 
in the art, such as vectors used in video encoding in order to reduce the bandwidth by 
reducing duplicity of the pixels. Therefore, it would have been obvious to one of 
ordinary skill in the art at the time the invention was made to modify Kaiser by using a 
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mathematical representation in order to save bandwidth and process the information at 
the client side. 

Regarding claim 49, Kaiser teaches a back channel (fig. 1, label 1400-data 
network). 

Regarding claim 50, Kaiser teaches product selection or information (col. 3, II. 3- 
5), collecting user information (col. 14, II. 3-21), and accepting information regarding a 
commercial transaction and completing the transaction (fig. 9). 

Regarding claim 51, Kaiser performs a 'secure transaction verification* procedure 
(fig. 9, label 9300), which in the broadest reasonable sense has some information 
regarding a viewer identifier in order to identify the viewer. 

Regarding claim 52, Kaiser teaches plural annotations, which as discussed in 
claim 28 have equal timing. Further, Kaiser teaches that the plurality of annotations 
would be different in that the system will need to differentiate among plural annotations 
in that they will have different masks and provide different information. 

Regarding claim 53, Kaiser teaches a reproducing apparatus (fig. 1, label 1300), 
which reads on a receiver that receives video (col. 5, II. 10-31) and action resource data 
(claimed annotation data) (col. 5-6, II. 55-8), wherein the action resource data includes 
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overlay information as shown in figure 6B (col. 10, II. 9-41), selectable actions (claimed 
object data appearing on a portion of the video) (fig. 6A-6D), and timing information (fig. 
2, col. 6, II. 9-64). Whereas Kaiser does not explicitly teach a data store storing at least 
a portion of the received annotation data, the system inherently stores a portion of the 
data in buffers, memory, or processor registers in order to process the received 
information. 

Kaiser teaches placement zones, which can be viewed over one or more frames 
(col. 6, II. 19-23), fails to explicitly disclose drawing the graphics images in response to a 
viewer request and the drawing of the graphic image being synchronized to the 
underlying frame based on timing data. 

In analogous art, Wistendahl teaches drawing halos H (see fig. 7a) by 
highlighting the hotspots (col. 15, II. 1-6, col. 15, II. 54-56), wherein the hot spots are 
viewed over one or more frames and the graphic image is synchronized to the 
underlying frame based on timing data of the graphic image, in that the hot spot is for a 
given frame and each frame has a unique time position in the sequence (col. 5, II. 67-3, 
col. 6, II. 42-46). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Kaiser by drawing the graphics images in 
response to a viewer request and the drawing of the graphic image being synchronized 
to the underlying frame based on timing data as taught by Wistendahl in order to 
provide for synchronization of the overlays over a plurality of frames, thereby increasing 
the effectiveness of the hotspot by properly following the image. 



1 
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Regarding claim 54, as shown in figure 6B, Kaiser visually identifies the object. 
Regarding claim 55, as shown in figure 6B, Kaiser teaches the location of the 

object. 

Regarding claim 56, as shown in figure 6B, Kaiser teaches shape information. 

Regarding claim 58, Kaiser teaches in-band techniques for receiving annotation 
data (col. 5, II. 45-54, col. 6-7. II. 65-4). 

Regarding claim 59, Kaiser teaches annotation data over a broadcast 
transmission with the video stream (col. 5, II. 45-54). 

Regarding claim 60, Kaiser teaches sending the annotation data over a 
broadcast transmission with the video stream (col. 5, II. 45-54), which clearly would be 
received prior to the video stream in order to synchronize the display of the annotation 
data. 

Regarding claim 61, Kaiser teaches sending the data over a separate 
transmission medium from the video (col. 5, 11. 45-54), which equates to an out of band 
signal. 
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Regarding claim 62, Kaiser teaclies sending the data over a data communication 
network (col. 5, II. 45-54). 

Regarding claim 63, Kaiser teaches permitting the user to engage in purchasing 
an item (col. 12, II. 29-65). 

5. Claim 57 is rejected under 35 U.S.C. 103(a) as being unpatentable over U.S. 
Patent 6,615,408 to Kaiser et al. and U.S. Patent 6,496,981 to Wistendahl et al. 
(Wistendahl) in view of U.S. Patent 6,415,438 to Blackketter et al. 

Regarding claim 57, Kaiser teaches displaying the annotations, but both Kaiser 
and Wistendahl are silent on second timing information for removing the annotation data 
from the data store. Blackketter teaches expiring triggers (col. 3, II. 13-22, col. 10, II. 24- 
35). Therefore, it would have been obvious to one of ordinary skill in the art at the time 
the invention was made to modify Kaiser and Wistendahl by indicating an expiration 
time in order to ignore invalid triggers (Blackketter: col. 10, II. 24-35). 

6. Claim 64-67, 71 , 72, and 74 is rejected under 35 U.S.C. 103(a) as being 
unpatentable over U.S. Patent 6,496,981 to Wistendahl et al. (Wistendahl) in view of 
U.S. Patent Application Publication 2001/0023436 to Srinivasan et al. (Srinivasan). 
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Wistendahl teaches a video source (fig. 1,2) and a converter for converting the 
media into a digital format of frames (col. 5-6, II. 60-2). Wistendahl teaches, as shown 
in figure 2, an N data defining hot spots (col. 6, II. 60-63), which is derived from an 
annotation source providing graphics data for overlaying a graphics image on a video 
frame (col. 6, IL 27-31 , col. 6, II. 42-46), wherein the hot spots are associated with a 
frame of the video, which reads on a data packet stream generator. Wistendahl 
teaches a server for transmitting the frame data (broadcast signal) and the N data 
(annotation data for the data packet stream generator) (col. 7, II. 42-45). Wistendahl 
teaches a converter (fig. 1, 2, col. 5-6, II. 60-2) providing timing information for the 
frames to the data packet stream generator in order to enable the data packet stream 
generator to associated the timing information to the graphical hot spots (col. 6, II. 27- 
31 , col. 6, II. 42-59, col. 6, II. 60-64). After formatting the data, the server transmits the 
data to the receiver (col. 7, II. 42-45). At the receiver, Wistendahl teaches drawing 
halos H (see fig. 7a) by highlighting the hotspots (col. 15, II. 1-6, col. 15, II. 54-56), 
wherein the hot spots are viewed over one or more frames and the graphic image is 
synchronized to the underlying frame based on timing data of the graphic image, in that 
the hot spot is for a given frame and each frame has a unique time position in the 
sequence (col. 5, II. 67-3, col. 6, II. 42-46). 

Wistendahl teaches a video source (fig. 1,2) and a converter for converting the 
media into a digital format of frames (col. 5-6, II. 60-2) and Wistendahl recognizes that 
the system can be used with MPEG-2 video, such as used on DVDs (col. 16, II. 42-65), 
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but Wistendahl is silent on the video encoder producing a transport stream. Official 
Notice is taken that the use of an encoding video into a transport stream is well known 
in the art. Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Wistendahl by encoding video into a transport 
stream in order to compress a digital signal by reducing the redundant information 
thereby reducing the bandwidth of a signal. 

Wistendahl teaches a separate (logical or physical) storage location for the frame 
data and the N data (col. 6, II. 39-59), but is silent on a data packet stream generator 
producing encoded data packets. Further, Wistendahl teaches a server for transmitting 
the frame data (broadcast signal) and the N data (annotation data for the data packet 
stream generator) (col. 7, II. 42-45), but is silent on a multiplexer. 

In analogous art, Srinivasan teaches a data packet stream generator producing 
encoded data packets creates metadata from the frame number or numbers (PTS), 
where the PTS is the presentation time stamp of a digital signal (pg. 19, para. 0222), 
wherein the PTS is associated with the metadata (pg. 21 , para. 0233), which equates to 
encoding data packets. Additionally, Srinivasan teaches multiplexing the metadata with 
the video content (pg. 20, para. 0224). Therefore, it would have been obvious to one of 
ordinary skill in the art at the time the invention was made to modify Wistendahl by 
producing encoded data packets using a data packet stream generator and multiplexing 
the video content and metadata as taught by Srinivasan in order to provide a digital 
synchronization means utilizing the existing structure of the digital signal and efficiently 
providing both the metadata and video to the user. 
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Regarding claim 65, the combination of Wistendahl and Srinivasan teaches 
timing information comprising one of a timestamp, timecode, frame numbering, or global 
time of day (Wistendahl: col. 5-6, II. 66-3, col. 6, II. 27-31, col. 6, II. 60-63). 

Regarding claim 66 and 67, the combination of Wistendahl and Srinivasan 
teaches a placement zone which references a product being displayed, wherein the 
placement zone is a location of an object (Wistendahl: col. 5-6, II. 66-3, col. 6, II. 27-31, 
col. 6, 11.60-63). 

Regarding claims 71-72, the combination of Wistendahl and Srinivasan has been 
explained above. Wistendahl teaches location and shape information of an overlay 
which can be highlighted (col. 6, II. 27-31, col. 6, II. 60-64, col. 15, II. 4-7, col. 15, II. 53- 
56). 

Regarding claim 74, Wistendahl and Srinivasan are silent on a mathematical 
representation of set of pixels. Official Notice is taken that mathematical representation 
of pixels are well known in the art, such as vectors used in video encoding in order to 
reduce the bandwidth by reducing duplicity of the pixels. Therefore, it would have been 
obvious to one of ordinary skill in the art at the time the invention was made to modify 
Wistendahl and Srinivasan by using a mathematical representation in order to save 
bandwidth and process the information at the client side. 
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7. Claims 68-70, and 73 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over U.S. Patent 6,496,981 to Wistendahl et al. (Wistendahl) and U.S. Patent 
Application Publication 2001/0023436 to Srinivasan et al. (Srinivasan) in view of U.S. 
Patent 6,615,408 to Kaiser et al. 

Regarding claims 68 and 69, the combination of Wistendahl and Srinivasan has 
been explained above. However, Wistendahl and Srinivasan are silent on a reference 
that represents a fixed relation to pixels, wherein the pixel is the upper left most pixel. In 
analogous art, Kaiser teaches placement zones for various frames (fig. 2, col. 6, II. 9- 
17), wherein the placement can track an inriage at the upper left corner (col. 10, II. 34- 
38), which equates to a reference that represents a fixed relation to pixels, wherein the 
pixel is the upper left most pixel. Therefore, it would have been obvious to one of 
ordinary skill in the art at the time the invention was made to modify Wistendahl and 
Srinivasan on a reference that represents a fixed relation to pixels, wherein the pixel is 
the upper left most pixel as taught by Kaiser in order to properly track an image thereby 
increasing the effectiveness of the placement zone. 

Regarding claim 70, Wistendahl, Srinivasan, and Kaiser are silent on a location 
reference at the centroid pixel. Official Notice is taken that a center position is well 
known in the art, such as tracking an object by the center position. Therefore, it would 
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have been obvious to one of ordinary skill in the art at the time the invention was made 
to modify Wistendahl, Srinivasan, and Kaiser by using a center position (as a reference 
point) in order to properly track an image thereby increasing the effectiveness of the 
placement zone. 

Regarding claim 73, the combination of Wistendahl and Srinivasan teaches 
location and shape information of an overlay which can be highlighted (Wistendahl: col. 
6, II. 27-31, col. 6, II. 60-64, col. 15, II. 4-7, col. 15, II. 53-56), but is silent on the an 
outline of the object. In analogous art, Kaiser teaches location and shape information 
such as the shape and location of the car (col. 10, II. 34-38), see figures 6A-6D, wherein 
the visual highlight (fig. 6B, label 6500) is a graphical overlay and has an outline of the 
car. Therefore, it would have been obvious to one of ordinary skill in the art at the time 
the invention was made to modify Wistendahl and Srinivasan by providing an outline of 
the object as taught by Kaiser in order to clearly convey to the user which object is of 
interest. 

Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Andrew Y. Koenig whose telephone number is (571) 
272-7296. The examiner can normally be reached on M-Th (7:30 - 6:30). 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Christopher Grant can be reached on (571) 272-7294. The fax phone 
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number for the organization where this application or proceeding is assigned is 703- 
872-9306. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto,gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 
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